Quantifying the Strength of Security Systems

Security researchers and practitioners lack techniques to quantitatively evaluate the strength of security systems against a determined attacker. Currently, evaluation is either qualitative, such as through security certification standards, or ad-hoc, such as through penetration testing and auditing. In this paper, we propose a framework that if applied to security systems, would produce quantitative measures that can be used to compare and appraise security systems relative to each other. Our framework utilizes public challenges in conjunction with an independent organization that mounts the challenges, regulates their implementation and certifies the results in an attempt to provide normalized measures. Unlike various ad-hoc challenges that have been run in the past, we believe our framework can create a quantitative, challengebased security evaluation infrastructure that is fair, sustainable and flexible.